Click any tag below to further narrow down your results
Links
Sean Goedecke argues that strong system design comes from deliberately dropping requirements like strict consistency or instant delivery, using examples from Unicorn, S3, StatsD, and SQLite. He also warns that coding agents tend to pile on unnecessary requirements that make code needlessly complex.
- Systems like Unicorn, S3, StatsD, and SQLite gain speed, simplicity, or scalability by giving up guarantees such as shared state, atomic multi-key updates, delivery of every packet, or concurrent writes.
- Deciding which requirements to relax needs both technical and business input, and trust gaps between engineers and product managers often block that tradeoff.
- Solo developers and startups can make these tradeoffs more easily because one person holds both the technical and business context.
- LLM coding agents tend to invent edge-case requirements (extra mutexes, TOCTOU checks, consistency guards), so reviewers need to strip them out or the codebase becomes unmanageable.
LinkedIn built Northguard and Xinfra to replace Kafka's core design after hitting limits at 32 trillion records per day. The new system decouples logs from replication units, distributes metadata across independent state machines, and lets new brokers serve traffic without copying historical data.
- Kafka's partition couples ordering, replication, and consumer parallelism into one heavyweight unit; moving or repairing partitions at LinkedIn's scale wastes network and disk bandwidth. Northguard uses smaller segments with independent replica assignments, so adding capacity doesn't require copying old data.
- The centralized metadata controller became a bottleneck managing millions of partition replicas. Northguard shards metadata across 128+ coordinators using a hash ring, with each vnode managing its own Raft state machine for topic, range, and segment metadata.
- Northguard can enforce fsync on all replicas before acknowledging writes while meeting LinkedIn's performance SLOs—a durability guarantee that Kafka's default configuration doesn't require because it relies on replication instead.
+ northguard
+ xinfra
+ kafka
+ linkedin
distributed systems
+ metadata management
+ storage architecture
This course covers core concepts of concurrency control—threads, locks, transactions, and crash recovery—in the first half, then shifts to distributed systems topics like network models, clocks, replication, consensus, and fault tolerance. It lists lecture topics, objectives, prerequisites, and recommended readings for a Part IB CST Michaelmas module led by Dr. Martin Kleppmann.
- Kleppmann's 16-lecture Part IB course splits evenly: 8 lectures on concurrency control, 8 on distributed systems.
- The concurrency half moves from basic threads/locks through bakery algorithm, semaphores, monitors, and message-passing (actors, CSP) to ACID transactions, 2PL, optimistic concurrency, and crash recovery.
- The distributed half covers network/failure models, physical and logical clocks, replication and consensus (Raft, FLP impossibility, 2PC, CAP), ending with case studies on CRDTs and Google Spanner's TrueTime.
- Core texts backing the course are Tanenbaum's Modern Operating Systems, Goetz's Java Concurrency in Practice, and Kleppmann's own Designing Data-Intensive Applications.
Martin Kleppmann discusses his journey from startups to academia, and the new edition of his book Designing Data-Intensive Applications. They cover trade-offs in modern infrastructure, cloud scalability, distributed system challenges, and emerging topics like formal verification and local-first software.
- Kleppmann rewrote and reorganized Designing Data-Intensive Applications' second edition to add consensus algorithm case studies and deeper cloud failure-handling coverage
- Cloud infrastructure has shifted the core scaling trade-offs from raw hardware capacity to network behavior and API guarantees
- Kleppmann is now researching local-first software algorithms for offline-first collaboration and cryptographic methods for supply-chain transparency that preserve pricing/manufacturing secrecy
- He predicts formal verification will become more important as AI-assisted coding introduces subtle bugs that are hard to catch otherwise
In this Pragmatic Engineer episode, Martin Kleppmann walks through updates in the second edition of Designing Data-Intensive Applications and shares how his LinkedIn experience shaped the book’s core concepts. He breaks down trade-offs in multi-region and cloud architectures, explains why replication still matters more than sharding, and predicts a rise in formal verification and local-first software.
- Second edition of DDIA cuts MapReduce coverage in favor of Spark and Flink, reflecting how the field has moved on
- Manual sharding has become rare thanks to bigger hardware and managed services, making replication the fault-tolerance skill every team now needs
- Kleppmann predicts LLMs will make formal verification mainstream by automating proof generation, shifting the bottleneck to human review
- Architecture decisions like multi-region or multi-cloud are business trade-offs between risk and cost, not universal best practices
Maintaining consistency in a system comprised of separate databases can be challenging, particularly in the absence of transactions. The article discusses the importance of defining a system of record versus a system of reference and emphasizes the Write Last, Read First principle to ensure safety properties like consistency and traceability in financial transactions.
- Write to the system of record last, but read from it first — this ordering prevents referencing data that hasn't actually been durably committed.
- If a downstream write (e.g., to a search index or cache) fails after the system of record succeeds, that's recoverable via retries; failing in the other order risks phantom reads of uncommitted state.
- Treating one database as the authoritative system of record (vs. other systems of reference) gives you a clear conflict-resolution and recovery strategy when multiple stores disagree.